Introduction to Time Series Analysis - 03

Introduction to Time Series Analysis - 03

This note is for course MATH 545 at McGill University.

Lecture 7 - Lecture 9

---

Test for weak stationarity

1. sample autocorrelation (for white noise only)

   problem: so many $h$'s

2. portmanteau test

   $Q = \sum^h_{j=1}\hat{\rho}^2(j)$

   If $y_{t} \stackrel{\text { iid }}{\sim} N(0, \sigma^{2})$, then $Q \sim \chi^2_h$

   $Q_{LB}=n(n+2)\sum^h_{j=1}\frac{\hat{\rho}^2(j)}{n-j}$

3. Turing points test

   $y_i$ is a turing point if $y_i < y_{i-1}, y_i<y_{i+1}$ or $y_i > y_{i-1}, y_i>y_{i+1}$

   Let $y_i$ be a turing point. For iid sequences, let $T$ be the size of turing points.

   What’s the probability of turing point after $t$? ANS: 2/3

   So $E(T)=(n-2)\frac{2}{3}$, $\nu(T)=\frac{16n-29}{90}$

   $\frac{T-E(T)}{\sqrt{\nu(T)}} \sim N(0, 1)$ for large $n$

4. sign test: count $y_{i+1} - y_i >0$

   Exact signal hypothesis test for $H_0: p=0.5$

5. Rank tests (compare ranks of $y_t$ with $t$)

(Note: there may be 20 minutes note that I did not take)

Prediction: $m(X_n)=E(X_{n+h}|X_n)$

Show that $E(X_{n+h}|X_n)$ is the unique minimizer of $E[(X_{n+h}-m(X_n))^2]$.

Assume $\hat{m}(X_n)$ minimizes $E[(X_{n+h}-m(X_n))^2]$, then $E[(X_{n+h}-\hat{m}(X_n))^2]$ is the minimum value of MSE.

$E[(X_{n+h}-\hat{m}(X_n))^2]\\=E\left[\left(X_{n+1}-E\left(X_{n+h} | X_{n}\right)+E\left(X_{n+h} | X_{n}\right)-\hat{m}\left(X_{n}\right)\right)^{2}\right]\\=\left.E\left[\left(X_{n+h}-E\left(X_{n+h} | X_{n}\right)\right)^{2}\right]+2 E\left[\left(X_{n+h}-E\left(X_{n+h}\right| X_{n}\right)\right)\left(E\left(X_{n+h} | X_{n}\right)-\hat{m}\left(X_{n}\right)\right)\right]+\left.E\left[E\left(X_{n+h} | X_{n}\right)-\hat{m}\left(X_{n}\right)\right)^{2}\right]$

Focus on the second term:

$2E[(X_{n+h}-E(X_{n+h}| X_{n}))(E(X_{n+h} | X_{n})-\hat{m}(X_{n}))]\\=2 E_{X_{n}} E_{X_{n+h} | X_{n}}\left[\left(X_{n+h}-E\left(X_{n+h} | X_{n}\right)\right)\left(E\left(X_{n+h} | X_{n}\right)-\hat{m}\left(X_{n}\right)\right) | X_{n}\right]\\=2 E_{x_{n}}\left[\left(E\left(X_{n+h} | X_{n}\right)-m\left(X_{n}\right)\right)\right] E_{X_{n+h} | X_{n}}\left[\left(X_{n+h}-E\left(X_{n+h} | X_{n}\right)\right)\right]\\=0$

So $E[(X_{n+h}-\hat{m}(X_n))^2]=\left.E\left[\left(X_{n+h}-E\left(X_{n+h} | X_{n}\right)\right)^{2}\right]+E[E\left(X_{n+h} | X_{n}\right)-\hat{m}\left(X_{n}\right)\right)^{2}]$

and this equation is greater than 0 unless $E\left(X_{n+h} | X_{n}\right)=\hat{m}\left(X_{n}\right)$ and then this equation equal to 0.

But this obviously extends to conditioning on $(X_1, X_2, ..., X_n)$

Choose a model for $E(X_{n+h}|X_n)$ and we always start from linear

What is the best linear predictor for $X_{n+h}$ that is a function of $X_n$?

$\begin{array}{l}{\quad E\left[\left(X_{n+h}-\left(a+b X_{n}\right)\right)^{2}\right]} \\ {=E\left(X_{n+h}^{2}\right)-2 E\left[X_{n+h}\left(a+b X_{n}\right)\right]+E\left[\left(a+b X_{n}\right)^{2}\right]} \\ {=E\left(X_{n+1}^{2}\right)-2\left(a E\left(X_{n+h}\right)+b E\left(X_{n+h} X_{n}\right)\right)+a^{2}+2 ab E\left(X_{n}\right)+b^{2} E\left(X_{n}^{2}\right)}\end{array}$

We take the derivative

$\frac{\partial}{\partial a}=-2 E\left(X_{n+h}\right)+2 a+2 b E\left(X_{n}\right)$

$\frac{\partial}{\partial b}=-2 E\left(X_{n+h} X_{n}\right)+2 a E\left(X_{n}\right)+2 b E\left(X_{n}^{2}\right)$

Set two derivatives equal to 0, we have

$\hat{a}=E\left(X_{n+h}\right)-\hat{b} E\left(X_{n}\right)=\mu(n+h)-\hat{b}\mu(n)$

Adding the equation above to this equation: $-E\left(X_{n+h} X_{n}\right)+\hat{a} E\left(X_{n}\right)+\hat{b} E\left(X_{n}^{2}\right)=0$

We have:

$-E\left(X_{n+n} X_{n}\right)+(\mu(n+h)-\hat{b} \mu(n)) E\left(X_{n}\right)+\hat{b} E\left(X_{n}^{2}\right)=0$

and thus $\hat{b}=\frac{E\left(X_{n+h} X_{n}\right)-\mu(n+h) \mu(n)}{E\left(X_{n}^{2}\right)-(\mu(n))^{2}}=\frac{\operatorname{Cov}\left(X_{n+h}, X_{n}\right)}{\operatorname{Var}\left(X_{n}\right)}$

So $\hat{a}=\mu(n+h)-\frac{\operatorname{cov}\left(X_{n+h}, X_{n}\right)}{\operatorname{Var}\left(X_{n}\right)} \mu(n)$

and $\hat{X}_{n+h}=\hat{a}+\hat{b} X_{n}=\mu(n+h)+\frac{\operatorname{Cov}\left(X_{n+h}, X_{n}\right)}{\operatorname{Var}\left(X_{n}\right)}\left(X_{n}-\mu(n)\right)$

If $\{X_t\}$ is stationary, then $\hat{X}_{n+h}=\mu+\frac{\gamma(h)}{\gamma(0)}\left(X_{n}-\mu\right)=\rho(h) X_{n}+(1-\rho(h))\mu$

Properties of $\gamma(h)$

1. $\gamma(0)=0$ (variance)
2. $|\gamma(h)| \leq \gamma(0)$ (Cauchy-Schwarz inequality: $|<u, v>|^{2} \leq\langle u, u\rangle \cdot\langle v \cdot v\rangle$)
3. $\gamma(h)$ is even: $\gamma(h)=\gamma(-h)$
4. $\gamma(h)$ is non-negative definite: $\sum_{i=1}^{n} \sum_{j=1}^{n} a_{i} \gamma(i-j) a_{j} \geqslant 0 \quad \forall n \in Z^{+}, \quad a \in R^{n}$

Even stronger property: let $\gamma(h)$ be a function defined on $h \in Z$, $\gamma(h)$ is non-negative and even $\Leftrightarrow$ It is the auto-covariance function of some stationary sequence

Strictly stationary series

Def. $\{X_t\}$ is strictly stationary if $\left(x_{1}, \ldots, x_{n}\right) \stackrel{d}{=} \left(x_{1}+n_{1}, \ldots, x_{n+1}\right) \quad \forall n \text { and } n$

Properties

1. all elements of $\{X_t\}$ are identically distributed
2. $(X_t, X_{t+h}) \stackrel{d}{=} (X_1, X_{1+h})$
3. If $E(X_t^2) < \infty$, then $\{X_t\}$ is weakly stationary
4. weakly stationary does not imply strictly stationary
5. IID process is strictly stationary

How to make a stationary sequence?

Let $\{Z_t\}$ be an iid sequence of random variables.

Let $X_t=g(Z_t, Z_{t-1}, ..., Z_{t-q})$, then $X_t$ us strictly stationary, because $\left(z_{t+h}, \ldots, z_{t+h-q}\right) \stackrel{d}{=}\left(z_{t}, \ldots, z_{t-q}\right)$

This sequence $\{X_t\}$ is q-dependent, i.e. $X_t$ and $X_s$ are independent if $|t-s|>q$

Generalize to weakly stationary, say that $\{X_t\}$ is q-correlated if $\operatorname{Cov}\left(X_{t}, X_{s}\right)=0 \quad \forall|t-s|>q \text { or } \gamma(h)=0 \quad \forall|h|>q$

Every second order weakly stationary process is either a linear process or can be transformed into one by substracting a deterministic component.

Def. $\{X_t\}$ is a linear process if $X_{t}=\sum_{j=-\infty}^{\infty} \psi_{j}-Z_{t-j}$ where $\{Z_t\}\sim WN(0, \sigma^2)$ and $\{\psi_{j}\}$ is a sequence of where $\sum_{j=-\infty}^{\infty}\left|\psi_{j}\right|<\infty$

We can view $\{X_t\}$ in terms of Backwards shift operator $X_t=\psi(B)Z_t$ where $\psi(B)=\sum^\infty_{j=-\infty}\psi_jB^j$. (Example: Moving average process has this property)

$E[|X_t|] \leq E[\sum^\infty_{j=-\infty}|Z_{t-j}\psi_j|] \\ \leq \sum^\infty_{j=-\infty}|\psi_j|E[|Z_{t-j}|] \leq \sum^\infty_{j=-\infty}|\psi_j|E[|Z_{t-j}|^2]^{1/2}$

Let $\{Y_t\}$ be stationary process with mean 0 and auto-covariance function $\gamma_Y(h)$. If $\sum^\infty_{j=-\infty}|\psi_j| < \infty$, then for $X_t = \sum^\infty_{j=-\infty}\psi_j Y_{t-j}=\psi(B)Y_t$, $X_t$ is also a stationary sequence with mean 0 and auto-cvovatiance funciton $\gamma_X(h)=\sum^\infty_{j=-\infty}\sum^\infty_{k=-\infty} \psi_j \psi_k \gamma_Y(h+k-j)$ where $Y_t = \sum^\infty_{j=-\infty} \psi_j Z_{t-j}$ and $\{Z_t\}$ is a White Noise process.

Proof. (There is a little difference in the notation of $k$ and $j$ because of different definition)

$\gamma_X(h) = E[X_t X_{t-h}] \\=E\left[\left(\sum_{k=-\infty}^{\infty} \psi_{k} y_{t-k}\right)\left(\sum_{j=-\infty}^{\infty} \psi_{j} y_{t-j-h}\right)\right] \\=E\left[\sum_{j=-\infty}^{\infty} \sum_{k=-\infty}^{\infty} \psi_{k} \psi_{j}\left(y_{t-k}\right)\left(y_{t-j-h}\right)\right] \\=\sum_{j=-\infty}^{\infty} \sum_{k=-\infty}^{\infty} \psi_{k} \psi_{j} \gamma_{Y}(h+j-k)$

If $\{Y_t\}$ is $WN(0, \sigma^2)$ process, then $\gamma_Y(l)=0, \forall l\neq 0$ $\Rightarrow$ $\gamma_X(h)=\sum_{j=-\infty}^{\infty} \psi_j\psi_{j-h} \sigma^{2}$

Example: AR(1) process

For $\{X_t\}$ stationary, let $X_t=\phi X_{t-1}Z_t$ where $\{Z_t\} \sim WN(0, \sigma^2)$. $\{X_t\}$ and $\{Z_s\}$ are uncorrelated for $s>t$

Define $\{X_t\}$ to be the solution to $X_t - \phi X_{t-1} = Z_t$

Consider $X_t=\sum^\infty_{j=0}\phi^jZ_{t-j}$, we have that $\{X_t\}$ is linear with $\psi_j=\phi^j \quad \text{for} j \geq 0$, and $\psi_j=0 \quad \text{for} j < 0$.

And $\sum^\infty_{j=-\infty}|\psi_j| < \infty$ iff $|\phi| < 1$

To solve $X_{t}-\phi X_{t-1}=Z_t$:

$\left[\sum_{j=0}^{\infty} \phi^{j} Z_{t-j}\right]-\phi\left[\sum_{j=0}^{\infty} \phi^{j} Z_{t-1-j}\right] = Z_{t}$

$\left[\sum_{j=0}^{\infty} \phi^{j} Z_{t-j}\right]-\left[\sum_{j=0}^{\infty} \phi^{j+1} Z_{t-(j+1)}\right] = Z_{t}$

$\left[\sum_{j=0}^{\infty} \phi^{j} Z_{t-j}\right]-\left[\sum_{j=1}^{\infty} \phi^{j} Z_{t-j}\right] = Z_{t}$

$Z_t=Z_t$

Therefore, $\{Z_t\}$ is stationary $\Rightarrow$ $\{X_t\}$ is stationary with mean 0 and auto-covariance function $\gamma_X(h)=\sum_{j=0}^{\infty} \phi^j\phi^{j+h} \sigma^{2}=\frac{\sigma^2 \phi^h}{1-\phi^2}$

If $|\phi| > 1$, then no stationary sequence exists that dependent on the past

Let $\Phi(B)=1-\phi B$ and $\Pi(B)=\sum^\infty_{j=0}\psi^jB^j$

Then $\psi(B)=\Phi(B)\Pi(B)\\=(1-\phi B)(\sum^\infty_{j=0}\psi^jB^j)\\=\sum^\infty_{j=0}\psi^jB^j - \sum^\infty_{j=0}\psi^{j+1}B^{j+1}\\=\psi^0B^0=1$

$X_t-\phi X_{t-1} = (1-\phi B)X_t=\Phi(B)X_t$

$\Pi(B)\Phi(B)X_t=\Pi(B)Z_t$

$\psi(B)X_t=\Pi(B)Z_t$

$X_t = \sum^\infty_{j=0}\phi^jB_jZ_t = \sum^\infty_{j=0}\phi^jZ_{t-j}$